[SPARK-6300][Spark Core] sc.addFile(path) does not support the relative path. #4993

pzzs · 2015-03-12T07:25:20Z

when i run cmd like that sc.addFile("../test.txt"), it did not work and throwed an exception:
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:../test.txt
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.(Path.java:172)
........
.......
Caused by: java.net.URISyntaxException: Relative path in absolute URI: file:../test.txt
at java.net.URI.checkPath(URI.java:1804)
at java.net.URI.(URI.java:752)
at org.apache.hadoop.fs.Path.initialize(Path.java:203)

merge lastest spark

AmplabJenkins · 2015-03-12T07:27:11Z

Can one of the admins verify this patch?

sryza · 2015-03-12T08:05:00Z

Thanks for taking this up @DoingDone9. Would it be possible to add a test for this?

sryza · 2015-03-12T08:08:07Z

core/src/main/scala/org/apache/spark/SparkContext.scala

    val schemeCorrectedPath = uri.getScheme match {
-      case null | "local" => "file:" + uri.getPath
+      case null | "local" => "file:" + file.getCanonicalPath


For scoping, probably better to move new File(path) into the match block.

Also, we should include the "file" scheme in the case as well.

pzzs · 2015-03-12T11:25:09Z

I have added the test, please test it. @sryza

srowen · 2015-03-12T15:15:00Z

ok to test

SparkQA · 2015-03-12T15:18:02Z

Test build #28516 has started for PR 4993 at commit 0ff3fa8.

This patch merges cleanly.

SparkQA · 2015-03-12T16:38:16Z

Test build #28516 has finished for PR 4993 at commit 0ff3fa8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-03-12T16:38:21Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28516/
Test PASSed.

srowen · 2015-03-12T17:34:18Z

core/src/main/scala/org/apache/spark/SparkContext.scala

@@ -1093,7 +1093,7 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
  def addFile(path: String, recursive: Boolean): Unit = {
    val uri = new URI(path)
    val schemeCorrectedPath = uri.getScheme match {
-      case null | "local" => "file:" + uri.getPath
+      case null | "local" => "file:" + new File(path).getCanonicalPath


I think this might be slightly more robust as new File(path).getCanonicalFile.toURI.toString. Then the "file:" can go away too.

SparkQA · 2015-03-13T03:53:07Z

Test build #28546 has started for PR 4993 at commit d594e16.

This patch merges cleanly.

pzzs · 2015-03-13T03:55:22Z

I have changed it. Please test it. @srowen

SparkQA · 2015-03-13T03:58:05Z

Test build #28547 has started for PR 4993 at commit ee375cd.

This patch merges cleanly.

SparkQA · 2015-03-13T05:14:47Z

Test build #28546 has finished for PR 4993 at commit d594e16.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-03-13T05:14:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28546/
Test PASSed.

SparkQA · 2015-03-13T05:18:38Z

Test build #28547 has finished for PR 4993 at commit ee375cd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-03-13T05:18:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28547/
Test PASSed.

pzzs · 2015-03-13T06:55:41Z

/cc @sryza @srowen

srowen · 2015-03-13T13:53:48Z

core/src/test/scala/org/apache/spark/SparkContextSuite.scala

  test("addFile works") {
-    val file = File.createTempFile("someprefix", "somesuffix")
-    val absolutePath = file.getAbsolutePath
+    val file1 = File.createTempFile("someprefix1", "somesuffix1")


This doesn't use the temp dir you created below.

srowen · 2015-03-13T13:55:17Z

core/src/test/scala/org/apache/spark/SparkContextSuite.scala

+        if (!gotten1.exists()) {
+          throw new SparkException("file doesn't exist : " + absolutePath1)
+        }
+        if (!gotten2.exists()) {


Same; feels like you can wrap up all these checks in a local function that you can just apply to both files rather than repeating code.

srowen · 2015-03-16T12:25:11Z

I think this is OK to go actually. The refactoring of the common code isn't a big deal, and, the temp dir issue is something I'm addressing in bulk in SPARK-6338 right now anyway, so I could touch it up along with that logical change.

…ve path. when i run cmd like that sc.addFile("../test.txt"), it did not work and throwed an exception: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:../test.txt at org.apache.hadoop.fs.Path.initialize(Path.java:206) at org.apache.hadoop.fs.Path.<init>(Path.java:172) ........ ....... Caused by: java.net.URISyntaxException: Relative path in absolute URI: file:../test.txt at java.net.URI.checkPath(URI.java:1804) at java.net.URI.<init>(URI.java:752) at org.apache.hadoop.fs.Path.initialize(Path.java:203) Author: DoingDone9 <799203320@qq.com> Closes #4993 from DoingDone9/relativePath and squashes the following commits: ee375cd [DoingDone9] Update SparkContextSuite.scala d594e16 [DoingDone9] Update SparkContext.scala 0ff3fa8 [DoingDone9] test for add file dced8eb [DoingDone9] Update SparkContext.scala e4a13fe [DoingDone9] getCanonicalPath 161cae3 [DoingDone9] Merge pull request #4 from apache/master c87e8b6 [DoingDone9] Merge pull request #3 from apache/master cb1852d [DoingDone9] Merge pull request #2 from apache/master c3f046f [DoingDone9] Merge pull request #1 from apache/master (cherry picked from commit 00e730b) Signed-off-by: Sean Owen <sowen@cloudera.com>

…ve path. when i run cmd like that sc.addFile("../test.txt"), it did not work and throwed an exception: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:../test.txt at org.apache.hadoop.fs.Path.initialize(Path.java:206) at org.apache.hadoop.fs.Path.<init>(Path.java:172) ........ ....... Caused by: java.net.URISyntaxException: Relative path in absolute URI: file:../test.txt at java.net.URI.checkPath(URI.java:1804) at java.net.URI.<init>(URI.java:752) at org.apache.hadoop.fs.Path.initialize(Path.java:203) Author: DoingDone9 <799203320@qq.com> Closes apache#4993 from DoingDone9/relativePath and squashes the following commits: ee375cd [DoingDone9] Update SparkContextSuite.scala d594e16 [DoingDone9] Update SparkContext.scala 0ff3fa8 [DoingDone9] test for add file dced8eb [DoingDone9] Update SparkContext.scala e4a13fe [DoingDone9] getCanonicalPath 161cae3 [DoingDone9] Merge pull request #4 from apache/master c87e8b6 [DoingDone9] Merge pull request #3 from apache/master cb1852d [DoingDone9] Merge pull request #2 from apache/master c3f046f [DoingDone9] Merge pull request #1 from apache/master

pzzs added 5 commits March 2, 2015 10:11

Merge pull request #1 from apache/master

c3f046f

merge lastest spark

Merge pull request #2 from apache/master

cb1852d

merge lastest spark

Merge pull request #3 from apache/master

c87e8b6

merge lastest spark

Merge pull request #4 from apache/master

161cae3

merge lastest spark

getCanonicalPath

e4a13fe

sryza reviewed Mar 12, 2015
View reviewed changes

pzzs added 2 commits March 12, 2015 17:34

Update SparkContext.scala

dced8eb

test for add file

0ff3fa8

srowen reviewed Mar 12, 2015
View reviewed changes

pzzs added 2 commits March 13, 2015 11:51

Update SparkContext.scala

d594e16

Update SparkContextSuite.scala

ee375cd

srowen reviewed Mar 13, 2015
View reviewed changes

asfgit closed this in 00e730b Mar 16, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-6300][Spark Core] sc.addFile(path) does not support the relative path. #4993

[SPARK-6300][Spark Core] sc.addFile(path) does not support the relative path. #4993

pzzs commented Mar 12, 2015

AmplabJenkins commented Mar 12, 2015

sryza commented Mar 12, 2015

sryza Mar 12, 2015

pzzs commented Mar 12, 2015

srowen commented Mar 12, 2015

SparkQA commented Mar 12, 2015

SparkQA commented Mar 12, 2015

AmplabJenkins commented Mar 12, 2015

srowen Mar 12, 2015

SparkQA commented Mar 13, 2015

pzzs commented Mar 13, 2015

SparkQA commented Mar 13, 2015

SparkQA commented Mar 13, 2015

AmplabJenkins commented Mar 13, 2015

SparkQA commented Mar 13, 2015

AmplabJenkins commented Mar 13, 2015

pzzs commented Mar 13, 2015

srowen Mar 13, 2015

srowen Mar 13, 2015

srowen commented Mar 16, 2015

[SPARK-6300][Spark Core] sc.addFile(path) does not support the relative path. #4993

[SPARK-6300][Spark Core] sc.addFile(path) does not support the relative path. #4993

Conversation

pzzs commented Mar 12, 2015

AmplabJenkins commented Mar 12, 2015

sryza commented Mar 12, 2015

sryza Mar 12, 2015

Choose a reason for hiding this comment

pzzs commented Mar 12, 2015

srowen commented Mar 12, 2015

SparkQA commented Mar 12, 2015

SparkQA commented Mar 12, 2015

AmplabJenkins commented Mar 12, 2015

srowen Mar 12, 2015

Choose a reason for hiding this comment

SparkQA commented Mar 13, 2015

pzzs commented Mar 13, 2015

SparkQA commented Mar 13, 2015

SparkQA commented Mar 13, 2015

AmplabJenkins commented Mar 13, 2015

SparkQA commented Mar 13, 2015

AmplabJenkins commented Mar 13, 2015

pzzs commented Mar 13, 2015

srowen Mar 13, 2015

Choose a reason for hiding this comment

srowen Mar 13, 2015

Choose a reason for hiding this comment

srowen commented Mar 16, 2015